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ABSTRACT 

This .paper describes the CURRENTLY OPERATIONAL version of the INGRES 
data base management system. This multi-user system gives a relational 
view of data, supports two higl^ level non-procedural data sublanguages 

i 

and runs as a collection of user processes on top of the UNIX operating 
system for Digital Equipment Corporation PDP 11/40, 11/45 and 11/70 com- 
puters. Stressed here are the design decicions and tradeoffs related 
to 1) structuring the system into processes, 2) embedding one comi]j;^nd 
language in a general purpose programming language, 3) the algorithms 
implemented to process interactions, 4) the access methods implemented, 
5) the concurrency and recovery control provided, 6) support for views, 
protection and integrity constraints and 7) the data structures used for 
system catalogs and role of the data iJase administrator. 
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.1 II.'THODUCTION 

lUGifLo (Interactive Grapnics and Retrieval Systen) is a relation- 
dl data base system ^nich is implemented on top of the UNIX 
operating system developed at Bell Telephone Laboratories 
[RITC74] for Dir^.ital Eqaipment Corporation PDF 11/40, 11/45 and 
11/70 computer systens* The implementation of INGRES is prinar***- 
ly prof^ranmed in "C", a hif^h level lan<^uage in which UNIX itself 
IS written. Parsin^^ is done with the assistance of YACC , a 
compiler-compiler available on UNIX [JOHN74]. 

Tne ddvantd/^es of a relational model for data base manap:ement 

systems have Deen extensively discussed in the literature, 

[CODD70,CODD74,DATE74] and hardly require further elaboration, 

I 

In choosinrj the relational model, we were particularly motivated 
oy (a) the hi^^h de'^ree of data independence that such a model 
affords, and (d) the possibility of providinp^ a hij^h level and 
entirely procedure-free facility for data definition, retrieval, 
update, access control, support of views, and integrity verifica- 
tion. 

In this paper we will describe the desif^n decisions made in 
lUGi^ES. In particular, we will stress the desif^n and implementa- 
tion of: 

a) the embeddinr: of all INGRES commands in the p;eneral purpose 
pror;ramminf^ lan^^ua^^e "C" 

b) tne access methods implemented ^ 

c) the catalofT structure and the role of the data bas^ adminis- 
trator 

-2- 



d) support for views, protectipr. and intej^rity constraints 

e) the decomposition procedure implemented 

f) implementation of updates and consistency of secondary indices 
recovery ana concurrency control 

Lxcept \>rhere noted to the contrary, this paper describes the 
INGRES system operational in January, 1976. 

To tn'id ^nd we first briefly describe in Section 1.2 the primary 
query langua,^^e supported, QUEL, and tne utility commands accented 
oy tne current system. The second user interface, CUPID, is a 
j^raphics oriented^ casual user lanr^ua;^e which is also ooerational 
and described in [!lCD075a, MCD075b].^t will not be discussed 
further m this paper. Then in Section 1.3 we describe the 
relevant factors in the UNIX environment which have affected our 
design decisions . 

In Section 2 v;e discuss the structure of the four processes (see 
Section 1.3 for a discussion of this UNIX' notion) into which 
IIIGREG is divided and the reasonin^: behind tne choice implement- 
ed. The EQUEL (Embedded QUEL) precompiler, which allows the sub- 
stitution of a user-supplied C program for the "front end" pro- 
cess is also discussed. This prop^ram has the effect of embeddinp; 
all of INGRES in the V.eneral purpose prof^ramminp; lanc^uaf^ie "C". 
Then ih Section 3 indicate the data structures which are Im- 
plemented in ItJQHES, the catalor, (system) relations which exist' 
and the role of tne data base administrator with respect to all 
relations in a data base. Tne implemented access methods, their 



-3- 

5 



calling corventions , and tne actual layout of data pac^es in 
secondary storage where appropriate are also presented. 

Sections ^, 5 and 6 discuss respectively tne various functions of 
eacn of the tnree Vcore" processes in the system; ' Also discussed 
are tne design and" implementation strategy of each process. 
Lastly, Section 7 draws conclusions, sup;gests future extensions 
and indicates the nature of tne current applications run on 
INGKES. 

1.2 QUEL AND TtiE OTHEH INGRES UTILITY COHMAWDS 

QUEL (QUEry Languaf;e) nas points in common with Data 
Lanf^uaf^e/ALPHA' [C0DD71J, SQUARE [DOYC73] and SEQUEL [bHAM71] in 
that it is a ^ complete [C0DDJ2] query language which frees the 
progranmer from concern for how data structures are implemented 
and what alrjorithms are operating; on stored data. As sach it 
facilitates a considerable det^ree of data independence {STON74a]. 

The QOtL examples in ,tnis section all concern the following rela- 
tion. . ■ 



NAME • DEPT SALARY MANAGER AGE 

Smith toy 10000 Jones ^ 25 

EIIPLOYEE Jones toy 15000 Johnson 32 

Adams candy 12000 Baker 36 

Jonnson toy UOOO Harding 29 

Baker adfein 20000 Harding 47 

Hardin,r; ajflmin 40000 • none 58 



,Indicated nere is at EMPLOYEE relation with dpmairja J4AME, DE^T," 
LiALAHY, MAiJAGEii and M5E. Each employee has a manarjer (except for 



Harding wrto is presumably the company president) , a salary, dn 

i 

age, and is in a department. 

A *QUEL interaction includes at least one RANGE statement of the 
form : 

BAIJGE OF variable-list IS relation-name 

The symbols declared in the range statement are variables which 
will be used as arf^uments for tuples.' These are called TUPLE 
VARIABLES. The purpose of thiSj statement is to specify the rela- 
tion over which each variable ranges. ' *!; 

Moreover, an interactiortx includes one or more, statements of the 

\ ^ ' 

form : \ 

Command [Result-name] ( Target-list ) 

[ WHERE Qualification ] 1 

Here, Command is either RETRIEVE, APPEND, REPCACE, or DELETE. 
For RETRIEVE and APPEND, Result-name is tne name of the relation 

« 

v/hich qualifyinn tuples will be retrieved into or appended to. 
For REPLACE and DELETE, Result-name is the name of a tuple vari- 
able which, throupih the qualification, identifies tuples to be 
modified or deleted • The Target-list is a list of the form 

Result-domain = Function... ^ 

Here, the Result-doma^i4:J^s are domain names in the result relation 
wnicn are to be as-signed tne value of the correspondinc; function. 



Tne following suggest valid QUEL interactions. A complete/ 

de^itriptior. of' the language is presented in [HELD75a]. 

' i 

Example 1 ; t'"'^-^;^;^^Fincl the birth year of employee Jones 

' ^' 'RAfi&E--9*^^-£U^ EMPLOYEe{" 

RETRIEVE INTO'W (BYEAR r 1-975 - E.AGE) 
• WHERE E.IIAflE = ''jQf.es''*' 



Here, E is a tuple variable^ s^^m^h ranges .over the EMPLCjfYEE ^ela- 
tior* and all tuples dn tnat r el aHqrr^ar^ found V^ich "satisfy^ the"" 

qualification .E.lAn^ -^r; "Jones". The result of tne quer/y is a ntew 

* \ ' \ I 

relation^ W,swnich Has a single ^'donain, BYEAR, that has been cal-; 

culkted- for each qualifying tuple. If the resii^t Relation is 

omitted., qualifying tuples' are written in display foVmat on th^ 

user's ^ te:^inin.al ?*or returned to a calling program in ^ prescrib.ed. 

format as discu^ssed in Section 2. Also,^in the Tarpe^^^^^t^^^^ 

"Result-domain .may be omitted if Function is the\right hand 

side is an existinfr domain (i.e. NAME = E.NAflE may be written as 



E.WAME ^ — see example 1.6). 



Example ].2 Insert the tuple t Jackson ,can-dy\ 13000 jBakei** , 30 X. 
into EMPLOYEE. * " *^ * " ^ * 

APPEND TO EMPLOYEe(NAME = Jackson", DEPT = "candy^j^ 
SALARY = 13000, IJGR '= "Bkker", AGE = -SOLo 



Here,'^tne result relation EMPLOYEE is modified ^by adding the 



indicated tuple to the relation. If not till domains are speci- 
fied, the remainder '.default to zero for numeric^ domains ahd null 
for character strings.- 

Example K3 ' If a. second relation DEPT(DEPT, FLOOR/0 contains 



tne floor* of each d'e|)ar,tment that an eraplo^^. might work in, 

tnen one can fire everybody or^the first floor as tS} 

. KAIWE OF E IS EMPLOYEE 
' RAliGE OF D IS DEPT 
DELETE E WHERE E.DEPT = D.'DEPT 
AND D. FLOOR* i 1 

/■*-■ ■ 




Here '-'E specifies that the EMPLOYEE relation is to be modified. 
All' tuples are to be removed which have a value for DEPT wliich is 
^tne sane as" some department of the first floor. 

Example 1.4 Give a 10 percent rai;3e to Jones if iie works on 
'the first floor 

RANGE OF E IS EMPLOYEE ' ' ' 

RANGE of D is DEPT - , . 

REPLACE ,E(-SALARY' BY 1.1 » E. SALARY) 
. WHERE B.NAME = "Jones" AND 

E.DEPt" = D.D'EPT AND D.FLOOR# = 1 

t 

Here, E.SALARY^is to be replaced by 1 . 1.»E. SALARY for those tuples 
in EMPLOYEE Where the qualification is true. (Note thatjthe key- 
words IS ^nd BY ^may used , interchangeably with "=:" in any QUEL 
statement . ) ^ ' • ^ 

Also, QUEL contains aggregation operators including COUNT, SUM,. 
MAX, MIN, and AVG. Two examples of the use of aggregation fol- 
low. \ . " 

Example I.5 Replace the salary of all toy department employees 
by the average toy^ department salary. 
RA,NGE OF E IS EMPLOYEE 

REPLACE E(S,ALARY BY AVG(E. SALARY WHERE E.DEPT = "toy") ) 
' WHERE E.DEPT = "toy" 



Here,. TtV6 — ia^^t^ be taker, of ttie salarv domain for tnoSe tuples 
satisfying* tne. qualification 'e*'DEPT = V'toy". Mote that 
AVG(E.3ALAKY WHERE^ c.DEPTs "t;oy") is scalar valued (in this in- 
stance, $13,000)"ar^---ean.seque^^ be called an AGGREGATE. 
More general agfjrenations are pos'slble as suggested by the fal- 
lowing example. • ' ^ ^' ' 

Example U6 Find those departments whose 'average salary 

exceeds the company wid4 average salary, both averages to be tak- 

"en- only for those employees whose salary expeeds $10000. 

RANGE Of E IS EMPLOYEE 
•R-ETRIEVE INTO HIGHPAY (-E.DEPT) 

WHERE- AVG(E. SALARY BY E. DEPf "W^IERE E. SALARY > 10000) 
> 

. ' AVG(E. SALARY WHERE E. SALARY > 10000) 

* 

Here, - AVG(E. SALARY BY -E.DEPT WHERE E. SALARY> 10000 ) is an 
AGGREGATE FUMCTIOIj and takes a value fo,r each value of E.DEPT. 
This value is the aggregate , AVG(E. SALARY WHERE E.SALARY> 10000 

Jp E.OEPT = value). (Fdr the toy, candy and admin departments 
is value is"respectiv"ely' U,500, 12,000 and 30,000.) The qual- 
ification expression for the statement is then true for depart- 
ments for which this a.ggregate function exceeds the aggret^ate 
AVG(E. SALARY WHERE E.SALARY>1 0000) . v 

/ ■ . 

n .addition to the above QUEL commands INGRES also supports a 
variety, of utility conmands These utility commands can be classi- 
fied in^o seven major categories. 

a) invocation of INGRES 
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INGRES data-base-narne 



Tnis command executed from UNIX "logs in" a aser to a given data 
base. (A data base is, simply a naned collection of relations with 
a given data oase administr'ator who Has powers not availab.le to 
ordinary users.) Thereafter, the user may issue all other- com- 
nands (except those executed directly fron UNIX) within' the en- 
vironnent of the invoked data base. 

b) creation and destruction of data bases 

I 

CKEATEDB data-base-name 
DESTROYDB data-base-r.ame 

These two commands are called -from Ufi^X. The invoker of CREATEDB 
must be autnorized' to.cr„eare data, bases Cln a manner to be 
de.^scribeS presently) .and "he autoraaticailly becomes the data base 
.--^^nistrator. DESTROYDB successfully •dest^'^y^ c>4ta base only 
if- invoked by the data base adninistrdtor . 

I 

c) creation and destruction' of relatfonT 

CREATE relnaraeC domain-name IS format, domain-name IS format,...; 
P'ESTROY relname 

■These commands create and destroy relations within the current 
data base. The invoker of the CREATE command becomes the, "owner" 
. oX the rplaition creat^ed. A user may only destroy a relation thut 

I 

he owns. I The current formats accepted by INGRES are 1, 2 and H 
byte inte-ers, 4 and 8 byte floating point, numbers and fixed 



lenr^tn ASCII character, strings between f jand 255/bytes. 

(1) bulk copy of data ^ \ 

COPY relnameC domain-name IS, format, dfomain-name. IS format,...) 
direction "filename" 

K 

PRINT relname 

The comnand COPY transfers an entire relation to or from a- UNIX 
file wftose name is "filename". Direction is either "TO" or 
"FfiOM". Tue fofmat for each domain i% a description of how it 
appears (or is to appear) in the UNIX file» The relation relname 
nust exist and have donain names identical- to the ones appearing 
in tne COPY cominand. However, the formats- need not arjree' and 
COPY will automatically convert data types. Also,, support is"' 
provided for dummy and variable length fields in a UNIX file. 

PRINT copies a relation onto the user's terminal formatting it. as 
d report. In this sense, it is stylized version of COPY. 

e) storage' structure modification*- [ , / 

nODIFY relname TO storage-structure ON (keyl, key2,...) 
INDEX ON relname IS indexname( key 1 , key2,.».f 

The MODIFY command' changes the storaf^e'''"^str|ucture of a relation 

from one aci^ess method to^ another. The five access methods 

currently supported are discussed /in Section 2. The indicated 

Keys .are' domains in relname v/hich are concatenated left to rintit 

f 

to form a combined key which is used in the organization of tu- 
pies 4.r. all^but one of the access methods. Only the* owner of a 
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relatipij. nay modify its storage structure. 




INDEX - cr|^a^s a secondary index for a relation . • it has domains 
of key'l', key2, . . . ,pointerT The domain, pointer, is the address 
or a tuple in the i^.dexed relation havinf; the, given values for 
Keyl, key2,.... An index named AGEI.NDEX for EMPLOYEE would be 
the following binary relation 



AGE POINTER 



^ 1 



^ . '25 address of Smith's tuple 

- 32 address of Jor^es' Jte^le 

AGEinDET^ 36 address of Adams" tuple 

29 address of Johnson's tuple 

^7 address of Baker's tuple 

58 address of Harding 's tuple 



Tne relation indexname is In turn treated and accessed just like/ 
any other relation except that it is aut<5raatically updated when 
the relation it indexes is updated. This is discussed further in 
Section 6. Naturally, only the owner of a relation may create 
and destroy secondary indexes for it, 

^f) consistency and integrity control . ; 

, INTEGRITY CONSTRAIMr is qualification 

., INTEGRI.TY CONSTRAINT LIST relnarae 

INTEGRITY CONSTRA-INT- OFF relname 

INTfiGRITY CONSTRAINT OFF (integer, ... ,inte(^er) 

RESTORE data-base-nanie ' • * 
s 

The first four command s . support the insertion, listinc^, deletion 

" * and selective deletion of integrity constraints which are to be' 

■'/;.. . ■ ■ ' 
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enforced for''all interactions with a relation. The mechanism for 
handling this enforcement is dlcussed in Section ^. Th^e last 
command restores a data base to a consistent state after a systen 
crast^^j^t must be executed from UWlX and its operation is dis- 
cussed . in Section 6. Tne RESTORiv^'^ommand is only available to 
tne data-b^se^ adninistrator. ^ 



JC 



^^ai^cellaneous 

HELP [relnane Imanual-se'ction] 
SAVE* relname UNTIL-.expiration-dat6 
RELKILLER data-base-name 

HELP provides information about the system or the data base in- 
voked. When called with an optional arr^uraent which is a^ connand 
name, HELP will return the appropriate page from the IHGRES 
reference manual [20pK75]» When called with a relation name as 
an argument, it returns all information about that relation. 
Wiphno argument at all it returns information about all rela- 
felons iri the current data base. 

SAVC is the mechanism by. which a user can .declare his , intention 
to keep a relation until a specified time. RELKILLER is a UNIX' 
command v;hich can be invoked by a data base administrator to 
delate all relations whose "expiration-dates'* have passed. This 
should be done when spa<?e in a data base is exhausted • (The data 
base administrator ' can ^also remove any relations from his da^a 
base using the DESTROY command, rep^ardless of who their owners 
are!) • • ' 



Two comner*j:s should be noted at this time. 



d) The system currently accepts tne language specified QUEL1 



in [HELD75a]. Extension is in progress to accept QUEL 



t>) The system currently does not accept views or protection 
sfeatenents. Although the algorithnis have^ been 'specified 
[STON74b,STON753 , they have not yet \been i^fplemen ted . For this 
reason, no syntax for these statenientsWs/^iven in this section; 
however the subject is dicus^ed further/ in Section ^. 

1.3 THE UtllX ENVIRONMENT / - ^ 

Two points concerning UNIX are worthy of mention in this section, 
a) The UNIX 'file system 



UUIX supports a tree structured file system similar to that of 
iiUIiS^ICS. Each file is either a directory (contai|iinp; references 
to descendant files in the file system) or, a data file. Each 
data file can be viewed as an array 1 byte 'p^lde and 2**2^1 bytes 
long. (It is expected that this^-m^jLmum length vill be increased 
by the UNIX implementors. ). Address.ing in a file '^is sinilar to 
referencing such an array. Physically, each file is divided into 
512 byte blocks (pages). In response to a' read i^equest., UNIJ( 
^ moves one or more pages from secondary memory to UNIX core 
buffers then returns to the user the the actual byte string 
ci^sired. If the same page is referenced^ apain (by-tche sam^ or 
another user) wnile it is still in a core buffer, no disk I/O 
tak^s pJ^ce., 

-13- . ' . 
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It IS important to note that UNIX pages data from the file system 
into and out of system buffers using a "least recently u^ed" 
replacement algorithm* In this way the entire file system is 
managed as a lartje virtual store* 

In part because 'the INGRES designers believe that a data base 
system snould appear as a user job to UIIIX and in part because 
they believe that fhe operating system should deal with all space 

■ ' 7 

manac^ement issues for the, mix of jobs beint; run, INGRES contains 
HO facilities to do its own memory management ♦ 

Each n.le in UWIX can be rjranted by its owner any combination of 
the following protection clauses: 

a) owner read ' ^ , 

b) owner vJrite - 

c) non^owner^read 

d) non owner write 

e) execute 

f) special exec'ute 

■ ■ -A ■. ■ 

When INGRES is lT.it ially generated, a UNIX user named INGRES is 
created • All data files managed by tiJii^^INGRES system are owned 
by this "super-user." •and-'^vp' their protection status set to 
"owner read, X)wner write, no othet? access" ♦ Consequently, only 
the INGRES super-user can directly tamper with INGRES files* 
(The protection sVsten is currently being altered to optionally 
require the consent of the data base administrator before unres- 
tricted access by the super-user is allowed*) 

The INGRES object code is stored in files whose protection status 
is set to "special execute, no other access"* * When a user 
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invokes the IHGRES system (by executing command a) above), UNIX 
creates ""the INGRES processes operating temporarily v/ith a user-id 
of IllGRES. When a user exics from INGRES these processes are 



desti/^oyed and the user is restored - to - -operafeing with his own 

/ ' - 

usernid. ^ . 

/ • ^ . «» , 

Using tnis mechanism, the only way a user may access an • INGRES 
aata base is to execute INGRES object code. This "safety lat&h" 
effectively isolates users from tampering directly with INGRES' 
data. * * J 

'b) The UNIX orocess structure 

A process in UNIX is an address space (6^K by.tes or iess on an 
11/40, 128k bytes or less on 11/i45's and 11/70's) which is asso- 
ciated with a user-id and is the unit of work scheduled by the 
UNIX sctieduler. Processes nay "fork" subprocesses ; consequently, 
a parent process can be the root of a process subtree. Further^ 
more, a process can request that UNIX execute a file in a'^'^desc'en- 
dant process. Such processes may communicate vith each other via 
an inter-process commut^ication facility called pipes". A pipe 
may be dedTar^d as a one direcftion communication link which Is 
written into by one process and read by a second one. UNIX nain- 
tains synchronization of pipes so no messages are lost. Each 
process has a "standard input device" and a "standard output dev- 
ice". These are usually the us^r 's' terminal but may bis redirect- 
ed by tne user to be files, pipes to otner processes, or other 
devices . 
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Lastly UNIX provides a facility for processes executinp; re- 
entrant code to snare procedure segments if possible. INGRES 
takes advantage this facility so the core space overjiead< of 
^^multiple concurrent users is only that reqj^lr£d--b^t-4at-a- segments-. 

— — . .^^-'^^ t 

We ^turn in the next section, to tne process structure in which 

* " *■ 

INGRES runs. 
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2' THE INGRES -PROCESS STRUCTURE • - . 

i ■ ' y * 

INGRES can be invoked^ in two ways: First, it can be .^directly 
invoked from UNIX* by executing INGRES data-base-name; second it 
can be invoked by executing? a pfrogram writtjen using the EQUEL- 
precompiler. Vie discuss each in /turn and tnen comment briefly on 
Why two mecnanisns exist. 

2.^ INVOCATION FROK,UNIX 

Issuing INGRLS as a UNIX comrtrand causes the process structure 
shown in Figure 1 to be created. 
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. ' ' - • ^; ■ ' ,. , ' ' 

Process 1 as an interactive terminal monitor which allows the 

user to formulate, print ^ edit and execute collections of INGRES 

commands. It niaintains a wor^jkjspace witi% which the user interacts 

until he is satisfied with'^is interaction.^ The contents of this 

v/orkspace are passed down pipe A as a string of ASCII char^acters 

When 'execution is desireid. ' . ' , ^ ^ 

■ ■■ ■ i. \ - 

As noted .above, UNIX allows a. user, to altB^r the standard input 

r 
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ar^d output devices for his processes when executing a iomman^. 
As a resulr^h'^ invoker qf INGRES may direct the terninal monitor 
tp take^ input 'from a \iser file (in which case he runs a "canned" 
tollection of interactions) and direct output to another device 
(such as the line printer) or a file. 

The current terminal monitor accepts th% following commands. 
Anything else is simply appended to the user's workspace. . ' 

# : Erase the previous character. Successive uses of this • 
instruction will erase back to, but not beyond, the 
bef;inning of the current line. -. 

§ : Erase the current line. Successive uses of this, in- 
struction are Ignored. 

\r : Erase the entire interaction (reset the workspace) . The 
former contents of the workspace are irretrieveably lost. 

\p ^: Print the current workspace. Its contents are pri/ited 
. on the user's terminaK, < j 

\e Enter Jbhe UNIX text editor and begin accepting editor 

commands. The editor allow? sophisticated editin^^ of the 
user's workspace. This cornm'and is executed by simply 
. "forking'' a subprocess and executing the UNIX editor in it 

\g : Process the current query (go). The content^ of the 
workspace are transmitted to process 2. ' '""^ 

\q. : Exit from INGRES. 



Process 2 contains a lexical analyzer, a parser, query modifica- 
tion routines for integrity control (and in the future support of 
views and protection) and concurrency control. V/hen process 2 
finishes, it passes a string* of tokens to prc^cess 3 through pipe 
B. Proce'ss ^ is di-^cussed in Section 

Process 3 accepts this token string and contains execution 
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routines for the commands RETRIEVE, REPLACE, DELETE and APPENC. 

Any update is turned into a RETRIEVE command 'to isolate tuples to 

be changed. Revised copies of modified tuples are spooled into a 

special file. This file is then^ processed by a "deferred update 

processor" in process ^ which is di^cus3ed in Section 6. 

/ 

Bdsically^ procejss 3 performs two. funSltionjs, for RETi^IEVE commands, 
a) A multiyaridble qCiery is DECOMPOSED into a sequence of in- 
teractions 'involving only a single variable, -d) A one- variable 
query is'executed ^y a one variable query processor (OVQP). OVQP 
In. turn performs it^ fUnctjLon by making calls on the access 
methods. 'These two functions are discussed in Section 5; the 
acqess raethod are indicated in Section -3* . ; 

In process 4 resides^ all code to support utility , commands 
(CREATE, 'DESTROY, INDEX, etc.). Process 3 simply passes to pro-. 



cess 4 any commands which ^ocess ^I'^ill execute. Process k is 
organized as a collection of overlays which accomplish the varl- 



in Section 6. 



ous; functions. The structure of this process will be discussed 

Error messages ar6 passed back throuj^h pipes D; E and ,F to. pro- 
cess ' 1 which ~ returns them to the user. If the (^<\iT\mand is a 

RETRIEVE with no result relation specified, process \\3' returns 
t \\ 

qualifying tuples in a stylized format directly to the \" standard 

* \ \ 

output device" of process 1. Unless redirected; tni^|^,Ud^ • tne 

useji's terminal. 
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./e now turr^ to the operation of INGRES when invoked by cc>d|p^\ Y^rom 
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tne precompiler. - i 

2*2 EQUEL 

AltnoUgh , QUEL 'alone provides the flexibility for most data 
management, requirements, there are^ many applications which re- 
quire a customized user interface in place of the QUEL languacje. 
For this as well as other reasons, it is often useful to have the 
flexitiility of a general purpose programming lan;^uar;e in addition 
to the data ^ase facilit'ies of QUEL. To - this end, a new 
lan(?uage, EQUEL (Embedded QUEL), has been implemented which con- 
^ sists, of QUEL embedded in the ger^^ral purpo.se prORramminR 
language "C". 

In this section v/e describe the EQUEL lahpiuage and Indicate, hov' 
it operates in the INGRES environment, ? ^ VJ?^ 

In the design of EQUEL, the following goals were set: . 

1) The new language must have the full capabilities of both "C" 
ana QUEL. 

* 

. 2) Tne' C program s/fould have the capability for processing each 
tuple individually which satisfies the qualification In a 
QUEL RETRIEVTstatement. (this iJs the "piped" return faoil-^'' 
ity described in Data Language/ALPHA [C0DD71D. 
3) Thfe implementation should make as much use as possible of 
the^ existing C and QUEL language^ processors, (ihe irnplemen- 
tation cost of EQUEL should be small). 

« 

Witn these goals in mind, EQUEL was defined as follows: 
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1) Any C languaf^e statement is a valid EQUEL statement* 

2) Any QUEL statement (or INGRES utility conmand) is a valid 
EQUEL .statement as long as it prefixed by two numbe^r 
sifjns, (•'•##"). 

3) C program variables may be used in QUEL statement's in place 

<6<3 

of relation names, domain nam^s , target listo elements, or 
domain values.' The declarat^ot^v^tatements of C .variablea.- 
used i^in this manner must also be ^pr^f i^ed by double number 
signs- 

4) RETRIEVE statements without a result r^lafi^n have the form 
RETRIEVE , ( Target-list) 

[WhSrE Qualification] iHK C-block #} 
wnichT results in the C-Block .being executeijl once for each 
qualifying tuple. • 

Two "Short 'examples illustrate EQUEL syntax. 

Example 2.1. The followlnp; section of cdde implements a small 
front end tQ** INGRES which performs only one, query. It reads in 
the name of an employee and prints out the employee's salary in a 
suitable format. It continues to do th-is-aa^long as ther^e are 
more names to ^b^y^ea^d : i^. . The functions READ and PRINT are as- 
sumed to have the obviqus meaning. 

main( ) • . . , 

{ • ' 

#// -cnar 1JAME[20] ; 
## int SAL; 
While (READ(NAnE)) 
- ■ ■ { . - • ■ ■ 

//# RANGC OF X IS EMP 

RETRIEVE (SAL = X. SALARY) 
##. WHERE X.NAME = I1AI1E 
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\ 

{ 

PRINTC'Tne salary of ",NAME," is ",SAL); 



In^ this example the C-variable NA^E is used in the ..cmalifi^a- 
^ tion 6f the QUEL statejnent and for each qualifying tuple, the. 
C-variable SAL is set to the ^ appropriate value and then the 
Print statement is executed, (note:, in C and are 

equivalent to telN and END in ALGOL). 




\ 

\ 



Read in a relation, name and two dorjiain names. Then 
ection of values^ which the ^corA domain is to 
essing on all values which |^he first domain 

I 

une the functions READ and PROCESS exist and 
nave the obvious meanings.) A more elaborate version of ^;hls pro- 
gram could serve as a slnipl\ report? generator. 

//# int VAI>UE; >^ \ i 

cMar-REL»AHE[ 13j , DOHaNAMk[ 1 3] , DOMVALCSO]; 
//# char D0MNAME_2[ 13J ; 
READ(HELNAME) ; 
1<EAD(D0MNAME) ; 
READCDOHNAME _Jil 
- "iNrmi^GE OF X IS RELNAME 

While (READ(DOHVAL)) • • ' 

{ ' • • 

RETRIEVE (VALUERS X.DOMNAME) 
## ■ WHERE X.D0M"fJAHEl2 = DOMVAL 

{ - V - 

PROCESS(VALUE) ; 

) ' ^ ' - 

, ) • _ 

Any RANGE declaration (in this case the one for X) is assumed by 
INGRES to Hold until redefined. Hence, only one RAIJGE statement 
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is required regardless of trie number of times the RETRIEVE state- 
ment is executed. , , ' 



In order to implement EQUEL, a translator ( pre-compiler) was 
written wnicn converts an EQUEL program into a valid C-prof^ram 
witn QUEL statements converted to appropriate C-Vd^ and calls to 
INGRES* The resulting C-program is then compiled by the normal 
C-compiler producing an executable module. Moreover, v/heif^/cin 
EQUEL program is run, the executable module produced by the C- 
compiler is used as the front end process in place of ttie 'ffc- 
teractive terminal monitor as noted in Fif^ure 2, 
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The Forked Process Structure 
Figure 2 



During execution* of fhe fr'ont-end program, data base requests 
{QUEL statements in the EQAJEL program) are passed throu^^h pipe A 

• \: , .-^^^ ' , ! 

ar*d processed 4^y' INGRES. If tuples must be returned for tuple at 
a tifte prpcessing, t^en.they are returned through a special data 
pipe*set uip. between process 3 and the C program. A condition 
c5cie "Is also returned through pipe F to Indicate success or the 

typ.e .of ■'errot; encountered. 

♦ * 
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Consequently, tne EQulL^^translator must perform the followinf? 
five -functions: 

I " 

1) i^.sert system calls to *j;spawn" at run time the process 

structure snoWn in Figure 2.« 

2) note C-variable declarations prefaced by/#// as le^^al for 
' 0 / 

inclusion in IHGRES commands. 

« . * 

"3). process other lines prefaced by H. These are parsed to 
isolate C-variable3. In adddition, C stj^ements are insert- 
^ ed to write the line down pipe A in ASCII format, modified 
so that values are substituted for any 1 "c^variabl es. The 
rationale for not completely parsing a QUEL statement in 
EQUEL is Riven in [ALLM76]. ' 

* H) insert, C statements to Vead pipe F for completion infor- 
mation and call tne procedure ilerror. The user may i defin 



Ilerror . himself or ^.ha'v^*^^" EQUEL include a' standard version 
Which print^s the error message (for abnormal terminations) 

' • " %' - 

and continues. , . ' • . , 

. \ ,1 

S) If data is to be returned thrcugi;! ' data pipe l(by a 

RETRIEVE with no "result relation specified?' EOUE|. ^must 

/ ^ • 1 ^ 

also: - 1 

■ ^ ^ - . . • 1 

a) insert C - statements to reajl the. data pipe for a 
t u^l e f o rma 1 1 ed as t y p e /-v al.u e pa^j^^v 



b) insert C statemen^^ to substit:»ute^ va^lues into C- 



%l1 

ar 355] 



3 

varl!S5les declared in the target 'list* If necessary, 

'* " , ' ^ 

values are converted to the types .of the declared C^- 
variables. " ' , , ' . 



c) insert C statements to pass control to the C-block 
•" ^ . . ^ 
following 'the RETRIEVE. 

' d) insert C statements fcj^llowing the block to return to 
step a) if there are more tuples* 

2*3 COMMENTS ON " THE PROCESS STRUCTURE 

The proce^ sAructur^ jshown i^nj^igures, 1 and 2 ig the fourth dif- 
ferent process structure implemented^ The following considera- 
tions^ sugg:ested this ^firial choice : 

V 

a) Simple control flow* Previous process structures' had a more 
fompleK interconnection of processes^hich made debugginr^ harder. 



b) Commands ^are--pra^sed to the right only* Process 3 must issue^ 
commantls to various overlays' in process 4 to execute-rfarrferactions 
as discussed in Section 5* Hence, process 3 must be/ to the left 
Of process 4. 

c) ' The utility commandsV are expected ^to be called relatively' 
infrequently compared fee the activity In process 2 and 3* Hence, 
it .appears appropriate to overlay- little used cpde in a single 
process* The alternative is to create addii/ional processes (and 
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pi^s) Which are quiescent most 'of the tine* This would require 



added space'^ih UUIX coreTtables for xo^ particular^ advantage < 



d) The first 3 processes are used^'frequently • Ovei^layiag code in 
these •processes was tried in a previous version and slowed tt\e 
system considerably • 

e) To run or. an 11/40, the Ji\K address space limitation must be 
adhered to. y-. Processes 2 and "3 are nearly their maximum size and 
nence cannot be^ combined . (For 11/45 and 11/70 versions- we may 
experiment with, such a- combination. ) ' . 

f) The C program v/hifch replaces the terminal monitor as a front 
end must run ' with a user-id different from that of INGRES for 
protection , reasons. -(Otherwise it could tamper directly with data 
managed by INGRES.) Henxje, either it must be overlayed into a 
process or run in its own process. For efficiency and conveni-, 
ence, the latter was chosen. 

g) The interactive terminal monitor coula have been -written 
(albeit clumsily) in «QUEL. Such a strategy would have avoided 
the existence of tuo process structures which differ only by the 
treatment of the data pipe. This was not done because response 

r * 

-time would have degraded and because EQUEL does type conversion 
to predefined types. This feature would unnecessarily complicate 
tne terminal monitor. 

h) The processes- are all synchronized (i.e. each^ waits for an 
error return from the next process to the right before cdntinuing 
to accept input* from the process to the left. This is done be- 
cause it simplifies the flow of control. Moreover in many in- 
stances 'the various processes IIUST be synchronized* Future 
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versions of INGRES may attempt to exploit ".parallel ism where pos- 
sible. - ", 




'3 DATA STRUCTURES_AND ACCESS METHODS 
Ue begin this sectiAr._.wJjtJi...a_dXsmssL 

manipulates and their contents. Then we sketch the language used 
to access all non directory files. Finally, the five possible 
file formats are indicated • 

'3.1 THE INGRES FILE STRUCTURE - . ' . 

Figure 3 Indicates the subtree of the UNIX file system; that 
INGRES rafanipulates. \ . 
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The INGRES Subtree 
Figure 3 



The root of this subtree is a directory made for the UNIX user 
^'INGRES". It has six descendant directories. The AUX directory 
contains descendant files containing tables which control the 
spawning of processes shown in Figures 1 and -2, and an authoriza- 
tion list of users who are allowed to create data bases. Only 
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tne INCHES "super-user" may modify these files (by using the UNIX 
editor). BIU and SOURCE are directories indicating descenclant 
flTes /of respectively <ytrj^c±uLarvd[ source code. TMP contains tem- 
files containing the workspaces used by the^Tnteracytive 
terminal monitor. DOC is tne root of a subtree with system docu- 
mentation and tne reference manual. Lastly there is a directory 
entry in DATADIfi for each data base that exists in INGRES* These 
directories contain tne data base files in a given data base as 
descendants. 

These data base files are of four types: 

a) "^n administration file. This contains the user-4.ct of the 
data base administrator (DBA) and initialization information. 

b) System relations." These relations have predefined names* and 
are created for every data base; They are owned by the DBA and 
constitute the system catalogs. ^ They may be queried by a 
knowledgeable user issuing RETRIEVE statements, however, they may 
bjet. updated only by the INGRES utility commands (or directly by 
the INGRES "super-user" in an emergency)* (When protection 
statements are implemented the DBA will be able to selectively 
restrict RETRIEVE access to these relations if he wishes.) The 
form and content of some of these relations will be presently 
d iscussed • 

) . ■ ' ' ■ . ■ 

6) DBA relations. These are relations owned by the DBA ar^d are 
snared in that any user may access them. When protection is 
implemented the DBA can "authorize" other users by inserting 



protection predicates (which will be in on^ of the system rela- 
tions) and "aeautnorize" them by <removins such predicates. 

d) Other relations. These are relations created by other users 



by KETRIEVE into ir'or CBEATE) and are NOT SHARED. ' - 

Three conraents should be pj^ade at this time. - ■ 

a) The DBA has the following power not available to ordinary 



users : 



M) the ability -to create shared relations and to specify 
access control for them 

2) the ability to run RELKILLER 

^ . 3) the ability to destroy any relations in nis data base 
(except the system cataloRs) 

This system allows "^^ne level sharing" in that only the DBA has 
the powers in a) and he cannot delegate any of these powers to 
others (as in the file systenis^of most time-sharing. systems)'* 
This, strategy v/as imf5lemented ' for thr^e reasons: 



1) Tne need for added generality was not perceived. 
Moreover, added- generality .would have created tedious 
problems (such as making revocation of acces,s privileges 
non trivial) .* i 

2) It seems appropriate to entrust to the DBA the duty 
^and power) to resolve the policy decision whicj)^'must be 
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made when space is exnausted -and some relation's must be 
destroyed (or archived ). Tnis policy decision becomes 
nuch 'Harder (or impossible) if a data base is not in tne 
control- of one user. " ^ . * , 

3) Someone must be entrusted with tne policy decision 
concerninr; which relations to physically store and which 
to define as "views". This "data base desi^^n" problem is 
best cen^tralized in a single DBA; 

b) Except for the single administration file in each data base 
ever/ file is treated as a- relatiol;. Storinc; system catalorts as 
relations has tne following advantages: 

r 

' 1) Code is economized by sharing routines for accesslnc^ 
bottf catalog and data relations. 

2) Since several storage structures are sapported for 
accessin^g data relations quickly and flexibly under vari- 
% ous Interaction mixes, these same storage Choices may be 



utilized to enhante access to catalofc information. 

I'. 3) The ability to execute QUEL statements to examine (and 

'% ' 

patch) system relations where necessary has greatly aided 
'^ system debugging. 

c) Ea\tft .relation is stored in a separate file, i.e., no attempt 
is made bo "cluster" tuples from different relations which may be 
accessed together on the same (or a nearby) page. This decision 
is based op'the,. following reasoning. 
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1) Tne access methods would tSe tnore compiroated if clus- 
tering were supported* T T 

2) UNIX has a small (512- byte) page size. -Hence it. Is 
expected that the number of tuples which, car/ be grouped 
on the sane page is small. Moreover., logically adjacent 
pages in a UHIX file are WOT ^^ECESSARIDY■ physically adja- 
cent. Her.ce clustering tuples on ">.earby'! pages has no 
meaning in UNIX; the next logical page in a file , may be 
further away (in terras of disk arm motion) than a page in 
a different%file. Jn keeping with, the de.sif^n decision of 
UOT modifying UKIX, these considerations were incorpprat- 
ed in tne design decision not to support clustering. 

3) Clustering of tuples only makes sense if associated 
tuples, can be linked together using "sets" [C0DA71] or 
"links" .CTSIC75]. Incorporating these access" paths into 
the decomposition scheme would hjave greatly increased its 
complexity. - . _ 

3.2 SYSTEM CATALOGS ^ 

We turn how to a discussion of the system catalogs. V7e discuss 

two relations in detail and indicate briefly the contents of , the 
others.. ' - \ • . > 

The RELATION relation contains one tuple for every relation In 
the data base (including all the system relations . ) The domains 
Of tnis relation are: * * . 
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spec 



indexd 



protect 

in teg 

save 

tuples, 

atts 

widtn 

prim 



tne narae of, the relation ^ / isji 

the UNIX user-id of the relation owner;, M 

^her. appended to.relid it produces a [ 
unique file name. for storinc; tne rela- 
tion ♦ ' ^ ] : 
indicates one of 5 possible storai^e 
schemes or else a i&pecial code indicating; 
a virtual relation (or "view"), 
flag se.t if secondary, index exists for 
this relation, (fhi^ s^Oag and , the i'ol- ; 
lowing two are present to improve, per for.- 

' mance by avoiding cat^og lookup's when 
possible during query modification and . 

^one variable query processing.) 
flag sejb if this relation has protection 
predicates. ' 



flag set i-f there are integrity con- 
-straints. - , * 

scrheduled life time of relation, 
number of tuples in relation.^ 
nupber of domains ir/ relation, 
width (in bytes) of a tuple. 

h ( ■ 

numt>er of primary file pages for this 
re; 



^* ^ 

elation . \y 



The ATTRIBUTE catalog contaips information relating to individpad 
domains of relations. Tuples of the ATtRIBUTE catalog contain 
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Lfte following itens for each domain of every relation in the data 



relid 
owner 

dona in- name 
donainno 



1^ 



name of relal^ion .in wnian attribute ap- 
pears " - 

relation owner 
doni'ain name / 
domain number ( position relation/ In 
processing interactions INGRES uses this 
number to reference this domain, 
offset offset in bytes from beginning of tuple 

to beginning o.f domain, 
^ype ^ data type of domain (integer, floating 

point or character string), 
length' lengtn (in bytes) of domain; 

keyno if this domain is part of a key, then 

"keyno" indicates the ordering of this ' 
domain within the key. 

These two catalogs together provide information about the struc- 
tuTB and content of each relation in ttie data base. Ho doubt 
items will continue to be added or deleted as the system under- 
goes further development. The first planned extensions are the 
minimum and maximun values assumed by the dom^n. -These will be 
used by a more' sophisticated decomposition scheme being 
developed ,*wnlch is discussed briefly in the next section and in 
detail in [WOr)G76j. Ttie representation of the catalogs as rela- 
tions has allowed this restructuring to occur very easily* 
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SeVeral otner system relations ejkist wnion provide auxiliary 
infornfatior. about relations. TrieN^NPEX catalog contains a -tuple 
for every secondary index in . tne data base. Since secondary 
indices are themselves relations tney are independently cataloged" 
in tne RELATION and ATTRIBUTE • relations. However, the INDEX 
catalog provides tne association between a priniary relation and 
the secondary indices for it ircludinR which domains/ of the pri- 
niary relation are in the index* 

Tne PROTLCTIOH and INTEGRITY catalor^s^ contain resjia^e^rfv 
protection and integrity predicates for eacti rfeiation in the data 
base* These predicates are stored in a partially^roce^ed form 
as character strings. (This mechanisni exists for INTEGRITY ar.d 
will be "implemented in tne' same way for PROTECTIOH.) The VIEW 
e^ataloj? will contain, for each virtual relation, a partially pro- 
cessed QUCL-.like description which can be used to construct the 
view from itk component physical relations. The use of these 
last three datalojjs will be described in Section 4. ♦ The ex- 
iste'nce of any of this auxiliary information . for a p;iven^ relation 
is signalled by the appropriate flaf^(s) in the RELATIOM cataloc^. 

Yet another Set of system relations ar^ thos'e used by the cjraph- 
ics sub-system to catalocr and process maps, which (like every- 
thing else) are stored as relations in the data base. This^ topic 
'has been discussed separately . in [G0753. 

3.3 ACCESS METHODS INTERFACE (AHI)» • / 

We will now discuss in raoi^e detail tne AMI wnicn Handles all 
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actual accessing of. data from relations. The AMI lanf^uage Is 
. implernerxted as a set of functions whose calling conventions are 
indicated below. 

Each access method must dq two things to support the follpwing 
calls. First it must provide SOME linear ordering of the tuples 
in^ aj relation so that the concept of "next tuple" is well de- 
'fined. Se.cond it n^ust assign to each tuple a tuple-id (TID) 
which un x^giial-y— i a tuple. 

The nine implemented calls are as follows: 

a) . openr(descriptO;r, mode> relation_name) 

Before a relation may be accessed it ,must be "^pened".' 'This 
.functior/' opens the ■ UNIX file for the relation and fills in a 
"descriptor" with information about the relation from ^ th^ 
RELATION and ATTRIBUTE catalogs. The descriptor, which must be 
declared in the callinpc program, is used in subsequent calls on 
Atil routines as an input parameter to indicate what relation is 
involved. Consequently, the AllI data accessinf: routines need not 
themselves check the system catalpgs for the description of a 
relation. "Mode" specifics i>/hether the relation is beinp^ opened 
for update or for retrieval only. ^ 

f> 

b) ;?et(descriptor , tid , limit_tid, tuple, next_flarj) 

.# , 

This function retrieves into [tuple ^ a single tuple from the 
^relation indicated by 'descriptoi^ \ 'tid ' and 'limit_tid ' are 
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tuple-identifiers. There are two modes of ret^rieval , "scan" and 
"direct". ^In "scan" node "get" 'is intended to be called succes- 
sively to retrieve all tuples within a range of tupre-id's. An 
initial value of 'tid' sets tne low end of the range desired and 
'liinit_tid' sets the high end. Each time "^etl' is called with 
•''next_f lag ' = 'TRUE, tne tuple- followincr 'tid' is retrieved and 
its tuple- id placed into 'tid' in readiness for the "next call. 
Reaching 'limit_tid' is indicated by a special return code. The 
initial setting of, 'tid' and 'liniit_tid' is done by the "find" ^ 
function. In '"direct" mode ( 'nex.t_flag ' = FALSE) the function 
re.T;rieves tne tuple, with tuple-id = 'tid '. 

o) find (descriptor, key, tid, match_mode) . ^ 

"Find" places in 'tid ' the tuple-id at the low or hif?h end of the* 
ranee of tuples whicti match the key value supplied. The matchini^ 
condition to beapplied depends on 'match-mode'. 

If the reldtion does not have a keyed storage structure or if the 
key supplied does not correspond to the correct key donains, the 
"tid' returned will be if no key were supplied. The objective 
of "find" is to restrict the scan of a relation by eiiminatin/; 
from consideration tnose tuples known from their ^placement in. the 
relation not to satisfy the matching' condition with the key. 
Calls to "find occur in, pairs, one to set the low end of a scan, 
the', other for the higjn end ,, and , the tv/o tuple-id 's 'obtained are 
used in subsequent calls on '*get". 

•^wo" functions' are available for determininf^ the access 



. ^ characteristics of the 'storage structure of a primary data r^la- 
tion or secondary index, respe'ctiveiy, " 

d) paramd(descriptor, access_characteristics_structure) . 

e ) parami(descriptor , access_characterist^cs_s£ruct uf»e) 

These functions fill in the 'access_characteristics_s^ructure ' 
wxth information regarding the type of key which may' be cq^-.- 

^ structed to optimize access to ^t\e given relation. This include3 
v/het her' exact key values or rarrges of key values can be used, and 

^ whether a partially specified key may bemused. This wil\ deter- 
mine the 'matcn-raode' used In a subsequent call to /'find" • The 
ordering of, domains in the key is also indicated. These, func- 
tions relieve optimization routines executed during the^ process- 
ing of an^/interaction of the need. to know directly about specific 
storage structures. ' ♦ . - 

Other AUl functions provide a facility for updating relations. 

D insert(descriptor ,. tuple)* ^ % ^ ^ \ ^ ^ 

The tuple is added to 'the relation in its "proper" place accord- 
.ing to it^s keV value anVl the storagev^ode of the relation* 

■ ; ^" • ' - 

.n). replace( descriptor, tid, hew_tupl§) 
* h) delete(descriptor , tid) 

The tuple indicated by. 'tid' is either replaced by new values or 
■ deleted from tne/ »^l^^tion altofrethe'r . The tuple-id of the af- 
^Vected tupie wilj,' have b^en obtained by a previous, "get". 



F^ihally, -when a.tl access to a relation is complete it must be"' 
closed : v ■ ♦ '* * , 

i) closer(descriptor) ^ ' * - 

. Ttiis ^lo^es tne relation's UNIX file and rewrites the information 
. in the descriptgu^ back into tne system catalof^s if therje has beef* 
• . any change. , v. ' . '5\ 

' ' * - • ■ * ^ , ' ' \ 

J.ii STORA^h^^TIiUCTUR'Es'^AlbABX;E; '% -N^-v.' o . /. 



Ue will now describe , the five storage structures; currently availr 
^ ' able in INGRES. Four of the schemes are -keyed, i^e., the sfeorapte' 
location of a tuple withiri tne file is a function of the v^lue of 
the tuplV's key domains. These schemes allpw rapid access 





sjiecific portions of <k relation when key values are supl 
^ -/^TeiJ^aininA i?;on-keyed scheme stories tuples in the file independent- 

Ij^es ar^ -^rovides,^ a low-overhead storap;e structure, 
e3^t5tf^ally attractive When th^, entire relation must be accessed 
anyway ; k ^ 1, ' . ' - . 

' ' . ' ' " ■ . • ♦ 

The non-keyed' stora^^e structure i%, INGRES is a randomly ordered . 
sequerftrai flle^ ' Fixed-Ien/^fi * tuples, are simply placed, 'sequen- 
tially* in» ^ the file in the order suppli'e^. jJew tuples added to 
the relation\'are merely appended to the end* of the fiMe, The 
^ unigue tuple-identifieE^or each tuple, is its byte-offset within 
the-f^le. This mpde is intena«d mainly f ' 

/ d) '^ery'^sma.! ] relations, for which tt^ overhead of other sohenes 
s unwarranfc'ed . * / , ' • 
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- b) transitional storage of data being moved into or out of thp 
systeni by COPYj 

. c') certain , temporary relations created as intermediate results 
during query processing. ^ i 

^In the remaining s-chemes, ttje . key- value of a tuple determines the 



.^e 




Aase offeUe file on whicti tne tjyple.*^ll be place.d. "(he' schemes 
sh-arg'-a common "p^ge-sfe^uot«r~e"»,f6r. iii-anaginR tuplejs on file -pa^es 
s showf. in ' Figu're' -4'.' , ' 
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A , tuple must fit entirely on a single' page. Its unique identif- 
•ler (TIB) consists, of a page. niJraber (the ordering of its page in 
the .UNIX file) - Plus a "line number" indicating its poMtion on 
the page. '^A "lite table", which grovS upwards from the ^bottom of 
the p^e, contains as an entry for each tuple', a pointer to tbe 
beginning ot the tuple. Ir, this way a page can be reorganized 
without affecting JID's: » ' 



Initially the file will contain all its tupl.es on ; a number o"f 
"primary" pages. If the relation grows and these pat^es fill, 
"overflow" pages 'are aJ.lp,cated ar,d chair^ed t?y pointers .to tne^ 
primary pages * with which .they are associated,. Within a chained! 
group of, pages no special ordering of tuples is Maintained. Thus 
in a keyed access which locate'? a particular primary page, tuples 

matching the key may \be on any, page in the chain* ' 

• <• 

As cliscus^ed in [HELD75b3 two modes of key-tp-address transforma- 

. - - ^ . . ^ ' ' . ^ ' ' 

tion are used — randomizinr^* and order p^eservin^^t |n ^a "hash" 
file tuples are bistriDiited randomly ^ throughout th^ , primary parses 
o'f the fl,le ^a9Cording to a ^hasEhing *.f unction on a key. This mod> 
is well suited for - situations in which'access is to be cofvdi- 
tioned vOn, a* specific key value. 

' y , ' ^ * ' • - . ^ 

/ks an order-preserving mode, a -scheme*' similar to IBM s ISAM 

' • -. / " - ■ . ' . 

[IBM66] is used. The' relation /is sorted to produce the order^g 
on a particular key. A multi-level 'directory is created wh-rSTi 
recortte-4;^he high .key on each| primary *p^ge. The directory,, wbi^ 
is static resides on several, pages within the ;fi^[^Jji,self^^--'f6l-• 
flowing -the primary pages. ^A primary page and Its* overflow panes 
are ' not maintained in sort order*. This decision is xli&cussed in 
the* sexi^tion oV. concurrency. The "ISAM-like." mode ^is useful ,in 
cases whe^re- the key value is likely to be specified as falling 

within k range of values / since a near ordering of* the /keys Is 

■ / — ' ' • ^ • - * ' ' " ^ / " • 

preserved.' The*, index c^pmpression sxiheme discussed in. X^ELDTSb] 



iS/ currently , under implementation. \ . / / 



' In tne above mentiorijed keyed modes, fixed lentrth tuples are 
/ stored. In addition, both' schemes can be used In conjunction 
with data compression T^ecHniques. [GOTT753. in cases where in- 
V creased srtorage utilization outweighs the^ added cost of. er-cpdinR 

acjjf^ decodi data during access. These modes ^ are Icnown as 

^ - , ] ' 4. ^ 

pre^^ssed hash" and* "compressed ISAM"* 

I*he current ^npression soh^rae suppresses, blanks and portions of 
^a tuple vmich match tn^^^^^^eding tuple. • This compression ^is 
.applied to, each ^j^^ independently.^ Other schemes are bein^r 
experimented witr^. " : ' . 

3.5 ADDITION OF HEW ACCESS METHODS , ' • ' 

One of the goals of the AMI design was to irjsulate higher level 
software from the actual functioning of the acce^^-lmethOds 4nd 
' thereby' to ^ake' it easy to add different ones. 

In order to add a new access'lnethod one need only extend the AMI 
routines to hancfle vhe new case> If the new rrfethod uses the same 
J. page layout and TID scheme, only find, parami, and*paramd need to 
be extended.. Otherwise ngj^procedures to perform these functions 
must ^ be m &0ljrgd""Tofu/e oy g§t,-' insert, replace and delete^ 



H THE STRUCTURC OF PROCESS Z ' ■ 

^ • 0 ;■ . 

Process 2 contains 'Cocle to perform four main functions . " ^ 

a) a le3!cical analyzer ' " . . " ' 

b) a parser (written in YACC [^JOHN?^]) , ' 

I • • • 

c) . qu,ery modification ro,utir*es tp support protection, yifews and 
integrity control - ' . 

d) concurrency < control ^ 



seo'are discussed in turn 



^•1 LEXICAL AHALYSIS AUD PARSING 



The lexical analysis and parsing phases of IHGRE^^ have- been or— 

^ r 

jjanized around fene YACC translator writing system available in 
OJWlX [JDHN7^].^ YACC takes a? input a description- of a .^ranmar 
consistinf^ of BNF-like parsing rules (^produoVi-ons) anq precedence 
rules, plus action statements associated with each production, 
^li^^pro^^ces a set of tables to b^ interpreted by a par3e table 
interpreter which is combined witti locally-supplie'^ ' lexical 
analysis and parsfnjE? action routines to produce" a complete trans- 
later. * / ' , ' 

The intferpr^eter uses a bottom-up LR(1) parsing approach. Th6 
lexical ana^lyzer is called to obtain successive symbols from the 
input stream as the inter pretep^^'attempts to rrtatcTT'Tjie input with' 
productions in the r^rammar. >^/hen a productior/ is matched YACC 
perforns a reduction and executes the action st4tement associated 
•with tne production* YACC has a mechanism for recoverinr, from 



errors to continue parsing? input in its entirety. , ; 

i/nile tne YACC parse , table interpreter checks the syntactic 
' correctness of tne input conmands, Uhe action statements check 

Tor sementic consistency and correctness and pr^^re the comrnands 

for further processin^^. Tne system 'catalogs are usQd.,' to check 
• rnat ^relation ahd doi^jgin n^iwes, formats, and so 'on , are- specified 

appropriately. ' , 

'.For utility commands a^coramand indicator and the parameters for 
the command are sent directly to* process 3 for transmission to^ 
process ^. Section J) discusses these commands and their imple- 
jjj§ntation» ^ ^ 

For QUEfL commands, the/anput is translated to a tree-striictjur^d 
ant^e^rnal form which idll be used in the remaining analysis and 
/ processing • Moreover, tne qualification part ^is converted to con- 
junctive norm^ad. form. The parse tree is now ready to undergo, 
wnat' has Xbeen ^termed ",query-rbodification to be described in 
section/4.2 and ^.,1. 




INTEGRITY 

Querg/; ■inodificati6n includes adding i^egrlty and protection- 

ficeciicateg to ttte original query aqd changing references to vir- 

tual relations into reference's to tne a-ppropriate physi/al rela- 
tdons.. At tne present time orsiy a— simple integrijiy scheme has 

- / ■ - ' ■* 

/ been, implemented. 

In [ST01J75] alRoritnms of several levels /oX complexity are 
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presented for performi|i^ integrity control or. updates* In %he 
present system only the .simplest case, involvinr^ sinr^le-variable , 
c^oK^-Sdte-free inte^^rity ^^assertions , nas been implemented and is 
described in detail in [SGii07'53. 

i>riefly., int>ef;rity a^^rttonjs ar-e entered in the form of QUEL 
quali fic^tior. ^^auses to be'applied to ir!teractions ^updatln^: the 
relation CJver wl^icn the yariaWe in the assertion ran^res* A 
parse cr^ee«.JfS created for the aualifi caption ^nd a representation 
of this tree stored in the IIJTEGRITi' catalor; tor^ether with an 
indication of the relation and specific domains involved. At 
query modification time, updates are checked' for any possible 
/ integrity assertions on the. affected domains* Relevant asser- 

tions are retrieved, re-built into tree form and f^rafted onto the 
update tree so as to AIID the assertions v/ith the existing qualif- 
ication of the interaction. 

i PHOTECTION- AfiD VIEWS o , ' 

Alf^oriChms' for the support of views ar-.e also given in [oTOf!7f3]. 
" ^ iiasically'- a' view is any. relation which cauld be created from 
existing reVatJons by the use of a'RETftlEVE command. Such view 
definitions will be 'tr'eated in a manner somewhat analogous »tp 
that used fDr int e;^ri^&y...j>on trol . They will be allowed in INGRES 
to support LCl|||^L programs written for obsolete versions of the 
data base and for user convenience^ ' 

Pr^teciion will be handled accor^zJ-inr; to the alf^orithm described 
in (STON74b3. Like intec^rity control this algorithm involves 

- p>' ■ 

er|c . 



adding qualifxcations " €o the user's^ interaction. Ir. t^e 
remainder of this section we df at ir.fruish t.hia protection sohenip 

* ~ * - - 

from the 6ne in ECHAflTS] and Indicate the "rationale behind its 
use . • • . " ^ 
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Consider the following two views: < , ' 

, RANGE PF E IS EMPLOYEE ' " . 

DEFINE RESTRICTION-I (E-.NAHE; E. SALARY, E.AGE) 
, WHERE £.DEPT = "toy" ^ • ' 

DEFINE RESrRICTI0N-2'(E.MAME, E.DEPT, E.SJ^LARY) " 
WHERE E.AGE < 50 

and the following two access control statements: 

RANGE OF E IS EMPLOYEE ' , " , 

PROTEC.T (E.NAME, E. SALARY, E.AGE) ■ 
WHERE E.DEPT = "toy" 

PROTECT (E. NAME, 'E. SALARY, E.DEPT) ' " - . 

WHERE' E.AGE < 50 

Acess control could be. based on vi6ws as suggested in [CnAM75] 

and a given user could be authorized to use views RESTRICT,I0N-1 

and ■ RESTl?ICTI0N-2. To find the salary of Harding he could inter- 

rogate EfESTRICTIOIJ- 1 as follows: ]. 

RANGE OF R IS RESTRICTION- 1 

RETRIEVE (R. SALARY) V/HERE ' 

fi.NAMC = "Harding" 



Failing to find Harding in RESTRICTION- 1 he would have to then 
interrogate RESTRICTION-2. . After two queries be would be re- 
turned the .appropriate salary if Harding was under 50 or in" -the 
■'-J' toy 'department .' 




I 

Under -the IHGRES scheme the user caf, issue 

BAHGE OF £ IS EMPLOYEE ' 
RETRIEVE (E. SALARY) WHERE 
•"E. WAME = "Harding" 

i , J * ^ ■ • ' . • 

__\ f • 

which' wil>-6e raodi"fi-ed> by the. access control algorithm to 

• ' X- ' ■ 

■ RAHGE qg, E .I^^^MPLOYEE . ^ . 

RLTRIEVE (E.^L)V^Y) VJHERE ^ 

E.IslAME = "Brbwn'''X : - . • • . -— ; 

AND / / ■ • . • . 

(E.AGE < 50 OR ^l.PEPT = "toy'O* ' 

In tnis way fcne user need not manually sequence throui^h his views 
to obtain desired data but ^ automatically obtains such /data if 
permitted* Uote clearly that the portion of EMPLOYEE to which 
th4 user has access . (ttie union of 'RESTRICTIOII-V and 
HE3TRICTI0IJ-2) is not a relation and hence cannot be defined as a 

S/ingle view. . ' 

I ^ . • ' 

'k) sjLinimdrize , access control restriqtions are handled autonati- 
cally by the IHGRES alf^orithms .but nrusU-^e d^alt with by a user 
sequencing throurth his views in a "view oriented'' accesjs cobtrol 
scheme. - ^ , 

^.^ CONCURREUCY CONTROL • 

In any .multiuser system provisions must!^ be included to ensure 
that . multiple concurrent updates are executed in a manner such 
that some level of data integrity can be guaranteed. The follow- 
inR twa (scnewhat facetious) updates illustr<ite the problem. 

'RAIJGE OF L is EIIPLOYEE 
U1 'REPLACE E(DEPT = "toy") WHERE - 

E.DEPT = "candy" 
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RANG|; OF E-.i^ EMPLOYEE 
R.B^LACE FCDEPT = "candy" ) "WHERE 
F.DEPT 5 "toy" 



JJ1 and U2 are "exec.uted concurrently with no controls, 
sone empl^y^^^ay end' up in' each department and the particular 
result may not be repeatable if the data base is backed, up' and 
fcne interactions reexecuted. C 

- - ' The control which nrust be provided is to {guarantee that 

sone data base operation is "atonic" (i.e. occurs in such a 

** f'^C^i*^'"' ^^^^ "it appears instantaneous and before or after any 
other data base operatit>nl.; Tjiis atomic unit Will be called a 
transaction. 

♦ - 

' In INGRCS there are three basic cljoices available for defir.infr.a 
transaction.* . * ' ' ' 



a) . something smaller than one IHGI|4irs cotnmand 



b) one IMGKES command 



c) a collection oV IIJ&RES commands ♦ 

If a) is chosen INGRCS could not /^uai^antee tiiat* tv/o concurrent^ 
executinf^ update commands gave the. same result as if they were 
executed sequentially (in either prder) ^n one collection of 
INGRES processes* In fact the outcome c'ould fail to be repeat- 
/kble, as noted-tr^^the example above. Thife situation is clearly 

I '"v , 

undesirable. 

)ptlor. c) 'is in the' opinion oC the INGRC's desi/jn.ers IniRpssib.le to 
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support. "The followinc transaction could be declared in ai^^EOUEL 



' prciffrani. 



BEQin TRANSACTION , . / 

IRST QUEL UPpATE • - , - . 

S^^TEM CALLS TO CREATE AND DEST-ROy FILES- ' • ■ 
SYSTEM CALLS 'TO POfiK - A 5EC09D.- COLLECTION OF INGRES 
* PROCESSES TO UHICIl QOmAHDS AREf PARSED 
~\SYSTEtJ CALLS TO READ FR^I A^ERMINAL 
. S-YSTEfl CALLS TO READ" FROM A TAPE 
SECdwii OUEL UPDATE ( wHose -forja depends or. previous\^tv/o 
systera^alls) ' ' - T 

EliD TRAHSACTI01-r\ ---- 






suppose T1 .is the above transaction and run.a concurrently with a 
transact!^ T2 involving commands of the same form". The second 
update - of each transjiction may v/ell "confrt^S^^ with the first 
update of the ocher* . note tnat .ttin^re p^s no way to t^H< aprlori 
tnat T1 arA T2 conflict because ttji^^orm of trie second update is 
[not knov/n \r, advance." Hence a deadlock situation can arise which^ 
can only be resolved by ^bortinrj te.r-e^^ansac.tion (an ur.desirc^bl 
;iolicy in the eyes of "tlie^NGREG /d 

out one Uran5action^*. The c^L^rhead of cs^ki^n^^^-^e^^^ t^rpuf^h the 
iocermediat/e System calls apoears prohi^bitive CiJ* rb^Is / possib.le 
at . all). ' kestricH/.n <4 tr^rjsaction. to have no sys^m cHjls ^ and 
hence no I/O) cripples "t^ie/pjbwer of a transaction in^rde^'to. 
make^ dej^dlock resolutio 
Thus' option t>) was 





The implementation of 

data "ibtems, pages, tupj/es|, domains ,^j;^lation 
by predicate locks .[ 
relatively crude phya 



sica.1 



etcT 
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X,^^!^^ or 
The cur?^t inpl^tnent^ion Is- by 



s (on domains 




.ioh) and > 
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'ax^plds deadlock' by r.ot allowing ar^interaction to proceed to pro- 

esources. Because of a 
jblera with" -tne cu>«ent desltyn fVKf»f>rtain access method calls,. 





The- Choice ofNijroidip/t deadlock rathei 
ing it is made priri^^Iy^ for iimplemei 
cttoioe of a crude iocking^ln^t^reflects a . miniconputer environ- 
ment wnere core storaj^e for a lahg^ loc^^Wl-e is not available. 
I.ni the future we plan to experimentsilly iniplement a crude and 



hereby low CPU overhead) version of a pred^^D^e locking scheme 



previously described in [STONT^o], Such an appro^cfrTnay ' provide 



considerable concurrency at an acceptable' overhead in lock table 
spac^ and CPU "time, although such a statement is highly specula- 
tive. • , . •* _ 



• Once the concurrency processor has assembled locks on all domains 
needed by an interaction, it may proceed to process '3 for unem- 
cumbered execution. " ' - 

To conclude this section we briefly indicate Jhe reasoninc behind 
not sortinr; a pef^te and. its overflow pafjes in the "ISAM-like." 

access raethod. This topi<is also discussed in [HELD75c]. 

\^ • ■ ■ ■ " • 

Basically,* maintenance of the soSs-^der of these pages may re- 
quire' the access method to lock more than one page when it in- 
■ serts a tuple. Clearly, deadlock might be possible given 



concurrent, updates and locks for physical paf^es would be reouired 



(at'' least once, a more sopnisticated predicate loekin^ scheme is. 
. ^ tried sucn^"<s^^TON74c] ) ^ To avoid both problem^ these pagBS 
remain unsorted^K^ the access methoKneed only be able to read- 



.podify-write a sinf^le page'^is an atomic operation* Althouf^li, such 
iin" 'atomic operation is not curT^.tly in UNIX (and. not needed by 
the current primitive scheme) it is^aNm.lnor addition. ^ 
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; 5 PROCESS 3 



-As noted ]in Section 2 this process performs . th^ followinn |wo 
functions which will be discussed in turn; 

a) ^ne DECOMt^OSITION of queries l^i^^^l^^^ 

into seqiaences of one-variable queriejs*. Partial results are 
'actjumulated until tne entire query is evaluated. .This profrram is 
o^led DECOMP. It also turns any updates into the appropriate 
to isolate qualifying tuples and spools new values into a 
speci'al file for deferred update. - ^ 

b) the processing of a single variable queries. The proi^ram is' 
calle^ the one jj^ariable query processor (OVQP). 




5/1 DECOUP 



Because INGRES allows interactions which are defined on the cross 
product of perhaps several rela.tions, efficient execution of this 
is of crucial* inportance in order to search ^s small a por- 
tion of the a^)propriate cross product sg^ace ^as possible.. ' DECOMP 
uses three tecliniques in processing Interactions . We -describe 
each technique then , give the actual algoritinn implemented* Fi- 
nally, we indicate tnq "hole of:a more saphlsti catted decomposition 
.scheme uricler desigi*^,» - ^^ • • ' . 



ii) Tuple substitution 

• % \ ^ ^ . V' ' • ^ . 

The basic technique used by DB60MP to reduce a query ♦ to/ fev/er 



variables - is t-^ple sub.3t itution • One variable (oixt oV possfbly 
*,uacy-)'*in ' tnfe > query is selected for si^bstituti6r,» Trie Afll 
.language is used* to scan tne relation associated with the vari- 



'>'iv^at)ii? orie Uiple at a time. For each tuple, the values of : donains 
' that relatrion are substituted into tne query. In tn*e result- 
• ing mcidifled query, a»ll previous references to the sub^stitutefi 
vt^aria^ie , have' nov/ been replaced by value's (consta^.ts) , and the 
i,*^ qqe'ry fhas thus b'een red'uced to one less variat)le. Precomposition 
. is *re.peated (recursively) on the nodified query unti.1, onlv one 



.y\fariaDl-e redains, at wtfWi point the pVQP,is called to continue 



viW^ point the OVQP is cal^d 
/ prbcessinf^. ■ :u • • ' 

One-Variable Detachment • 

If the qualiri caption C of iiie query is of the forn « ; 

• QUV1) Q2(V1, . ,Vn) ' ' 

for sotne tuple variable V'l , the follov/inrr tv;o steps can.be e:^e- 
cuted :' ' \ -^^ > 



. . 1) > Issue tne query ^ 1 

. ivETRIEVk: liJTO W (TL[V13) . - 

• u'liBHb 01[ V1j . : / . 

Here TLCvrj are" those domains required in the remainder of the 
query. . uote that this is' a one variable qjuery and^may bt^ passed 
directly tol OVQP. * ■ 

o^-i:) Heplaoe HI, the r.elatibn oyer *whicn VI rat/i^es, by W in A.he 



.rdn.p;e. declaration and, delete C*i[V11 from O.v 



Tne qu^ry fomied in 1) is called a "one- variable,, detachable- 
sub-query" (OyiXSQ) ar.d'the tecnniquevfor rornincr and executinfr it 
"ohe^-varia'ble detacbnent" (OVD). Tbis step" bas the effect of - 
reducing; tbe size of tbe relation over Wbicb VI ranges by ■ res- 
triction and projection. Hence, it nay reduce ^tbe complexity of 
tbe processing to follow. ' ■ " ' 



Iloreover, the opportunity exists in tbe process of creating new 
relations tbrougb OVD, to cboose storage -structurej "(and particu- ' 
larly keys) vbicb will prove helpful in furtber processing. 

c) Reformatting^ 

When a tuple variable is selected for substitution, a large 
nuiaber of queries aaoh witb one less variable will be executed. 
If b) is a possible operation after the substitution for sCme 
remaining' variable, Vl. tben tbe relation over wbicb '.VI rarrres. 
HI, can b6 reformatted to nave domains used in 01 (VI) as a key. 
This will expedite b) each time it is executed during tuple sub- 
, ^ stitution^ ^ , 

*^f^^^carN now state tne complete decornposition algorithm. 

a) If number of variables in query is 0 or'V call OVOP and \ 
stop; else go on. ' < . 



^ b) Find all , variables, ^ {VI,... Vn}', for vvhich the quer^y qontains 
a one-<rariable clause. Perform OVD to create new ranp;es for each 
of: tnese " var^iables. The new Relation for "each, variable , Vi , Is 

\ 
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stdred as a hash file with key,- Ki , chosen, as follows. 

' ^10 Tor each, j select from the renfrainln:<T ntulti-variable 
cl^alises the'query tne collectior., C(ij), wl?icn ' have 
tl3(? form • • : • ' 

■.V * ^\ y . 

' . . Vi.di r Vj^dj * \ / V 

/ wnere cli,cij are domains of Vi and V j • 
25 For.::) tne'l^ejr fCi to be* -the concatenation of - domains 
dil ^di2 , . • .Of Vi appearinfj in clauses in C(ij). 
3) If more than one j exists, for, which C(i,i) is non emp- 
ty, one"C(ij.) is chosen arbitrarily for forminr^ the key. 
If C(ij) i-s ergiity for all j'y.the relation is storecf as' an 
^ unsorted cable. • 

c) " Choose the variable , "Vs , with the snalle^^t number of tuples 
as the next one for which to perform tuple substitution i 

d) For each tuple variable Vj for v/hich C"(js) is non null, re- 

format tne storage structure of the relation Rj which it ranr^e-s 

ever, if r.ecesSary, so that' the key of Rj is tne concatenation of 

domains djl,..-. <appearin'f^ C(js). This ensures that when tfte 

clauses in C(js)> beqojne one-variable after substitu^tin^ for Vs> 

^subsequent calls to OVQP to restrict further the ranr^e of Vj will 

be^ done as* ef ficiently^^as possible. 

• 

e.) Perform tne /ollowinf; two steps for a.ll tuples i^i the ranq:e 
of the variat^le selecti^d in (c) : . \ 

1) ^i^bstitute Values from tup.le into query • / 

■ / ■ ■ * 
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2) call decomposition algorithm recursively on a copy of 
resulting ' query which no\^ has been reduced by one vari- 



able. 



Tne following comments on the algorithm are appropriate: 

a) OVD is' almost always assured of speedin^r processinf?. Not 
only- is it possible td wisely choose tne storai^e structure of a 
temporary relation out' also the cardinality of this relation may 
be much less than the one It replaces as the rant^e for a tuple 
• variable.. It only^fails if little or no reduction ' takes place 
^tjd-^ re formating'* is unphoductive. . '": 



It should be noted/that a temporary relation is created rather,, 
than a list of qualifying tuple;_-id 's . The basic tradeoff is that 
OVD must copy qualifying tuples but can remove 'duplicates created 
during the projection. Storinf; tuple-id's avoids th.e copy opera- 
tion at the expense of regcc^ssint^ quali,fying tuples and retain- 
ing duplicates. It is clear that cases exist where each stratep;y 
is superior. Tne INGRES designers have chosen OVD .because it 
does not appear to offer worse performance than the " alternative , 
allowi^ more accurate choice of the variable- with the snal-lest 
scange in step c) above and results in cleaner code. ^ ' 

\ 

b) Tuple, substitution is done when necessary on "the variable 
associated with ^he smallest' number tuples. This has the 
effect of reducir^fj the., number of eventual calls on OVQP. 

c) {Reformatting is done (if necessary)" with the knowlldRe tnat 

* / ,• , ■ o . • ' ' ' , 
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it will replace a collection of conplete sequential scans of , a 
relation by a collection of Timited scans. Tnis v/ill a.lnost 
always reduce processinr; tine. ^ - ^ 

d) It is ^believed tnat this alr^orltnm efficiently handles a 
larce class of interactions. Moreover ,* the alpjorithm does not 
require excess^^ve CPU overhead to perforn. _ There are, however, 
cases where a more eldOorate dl.f^orithm is needed, Th^. followin<^ 
coiTinent applies to these cases. ^ ^ 

/ * * ; 

e) Guppose that v/e have two or nbre strat^ies St/0, fJTl, ... 
,oTn, each one /beinr; D.etter than the previous one but also re- 
Quiriho; d 7^r^ei\.^T overhead. Suppose we bertin an interaction on 
oTO and rur. it f*or an amount of tine equal to a fraction of the 

e3timated c^verhead of oTI. At the end of tnat time, by sirapl; 

\ . ' ' 

counting the number of tuples of the first substitution varia^Sle 

i ... / 
which have already been processed, v/e* CjcAE. f^et an estimate P^r the 

total i^r(i£J^ssi^f; time usinr^ iiTO. If tni/^ls sir^nificant^ rrreater 

than the overhead of iiVl, then we switch to oT1. jStherwise we 

stay-'aild complete proc§3siJig_Uieinteraction usirjf STO. Obvious- 



ly, tne" procedure can b^.e repeated on Str to call ST2 if neces- 
sary, and so forth. . v - ^ • 

The alrjorithni detailecJ in this section is STO. A more sophisti- 
cated" dlriiorithn oTl is currently under development and is dls- 
'aus'sed in [WOiiuTOj. 

ofii: VAiaAifLij oui:hy PHncb\nson (ovop) 



Triis proftram is concerned solely with the efficient accessing of 

tuples from a single relation Riven a particular one-variable 

query* The initial portion of this program, known as STRATEGY, 

determines what key, (if any) may be profitably used to access the 

relation, ^what'' the value(s) of that key wil3, be used in calls to 

• "^he AHI routine "find", and whether the access nay , be aocom- 

plished directly through the Afll, to the storage .structure of the 
' . . * . \^ 'J ' 

^.iiklation ltaeliL,Qr^^if a_ sjseonda^^ry^ Index on the relation should be 

used. If cipcess is to be through a secondary index then STRATEGY 

must Choose which OWE of possibly many indices to use* 

/ ^ ' \ • 

, Then, the tuples retrieved according to the access straterjy 

selected are processed by the SCAM portion oP OVQP. This pror^ram 

evaluates eacn tuple against the qualification part of the query, 

creates tarr^et list valufes for qualifying tuples, and dlspo/es o.f 

.the target list appropriately. , < ' ^ 

Since SCMi is relatively straightforward, we discuss only the 
, policy decisions made in STRATEGY* 

First STRATEGY examines the qualification for clauses' which 
specjlfy the value of a domain , ^ i.e • clauses oT the form: 

/ ' '*V»domain op constant.* 

v/here /'op'' i? one of {r, <, <=,.> = }. . Such-clauses are termed 
^ "simple" clau3es and are orjf^anized into a list 
/ , • 

Obviously d non^.simple clause may be eouivalef*t to a simple one* 
For example * 
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E.SALniiY/2 =• 10000 is equivalent to E. SALARY = 20000. 
l{ov/ever, recoffnizing ar.d convertir.;t such clauses requires a r?er.-. 
eral algetfraic symbol manipulator. This issue has beer, avoided 
Dy ignoring al3 non-simple clauses. STRATCGY must now select one 
of two accessin';^ strategies: 

^a) is'suinr; two AMI .find comnands on the primary relation *fol~ 
lowed by a sequential scan of the relation between the linits 
specified 

?< . 

b) issuini^ tv/o AHI find commands on sone index relation followed 
by a sequential scan of tne i^-jdex between the limits specified. 
For each tupla retrieved the "poir.ter'V aonain is obtained and is 
a tuple- id of a tuple in the primary relation. This tuple is 
fetched ana examined. 

Key information about the primary relation is obtained usinr the 

i 

AHI function 'jj)iirdmd". Names of indices are obtained from the 

index cata'iop: and keyinf; information about indices in obtained 

I 

with tne t'unction "parami". 

STRATEGY now checks if a simple clause is available to limit the 
scan of the primary relation or an index . relation . If a relation 
is nasned the simple clause must spf^cify equality as the operator 
in-order to be useful. ISAM structures on the other hand allow 
ranges of values and less than all keys may be specified as lonn 
as the first one is present for structures with combined keys.. 

oTRATLGY checks for such a simple clause for a relation in t.ne 
follov/in<^ order: 



a. nashed primary relation, 
, b. nashed index ' ^ 

' \ c. ISAM primary* relation 
d. ISAM index 

The rationale for tnis ordering is related to the expected number 
of page accesses required to retrie\|e a tuple from . the source 
relation in each case. ^ . . " 

In case a) the key value provided locates a desired source tuple 
in one access, (ipjnoring overflows). In case b) the key value 
locates an, appropriate index relation tuple in one access but an 
additional access in required to"^)^trieye the proper source tu- 
ple. For tne ISAM scheme, the directory must be examined. The 
number of accesses incurred in this look*-up is at least 2. 
Again, with, an index, an additional access is required, making 
the total at least 3. ^"^^ case d). 



6 ■ UTILITIES 111 PROCESS n 
6.1 HlPLEilEUTATIOIJ OF UTILITY COflHANDS 

\}e nav.e indicated in Section 1 several data t>ase utilities avail- 
able to users. \;e will. row briefly describe their ifnplenentat ion 
and indicate their con'f i^uratjLo_n_Jo.thin the INGRES system. 

The comnands are organized into several .overlay pron;ran:s. Since 

a * \ 

an overlay may have more than one entry point, - it can contain 
mofe^ than' "one utilit"y command . In fact, the utilities are 
grouped where possible to minlnize overlayinrji* The overlays all 
contain a comnon main propiram known as the "-controller'* , which 
reads pipe C and writes (Completion messa^^es into oipe D. The 
processing of a utility command occurs as follows. 

First, the parser recof^nizes a utility command in a user interac- 
tion. This name is looked up in the INGRCS "process-table", 
wnich has an entry for edch command nane in the lanftua^re. Cach 
entry has an "overlay-id" and a "function-id". The first indi- 
cates the overlay program containinr^ the command, and, the second 
indicates the proper entry point within that overlay. 

These id's are passed down pipe B to process 3 .followed by the 
parameters of the command specified* Process 3 determines Trom 
tne ''overlay- id" if the command is a utility command intended for 
process If 30, ttie inforraation is simply written on pipe C. 

At this point, some overlay is occupying; -process 4, having 
remained froM a previous command. Its Copy of the controller 




reads pipe C to obtain the overlay-id. If different overlay 
from me present one is indicated, the' ciontroller overlays pro- 
cess ^4 and control passes to the pcntrollel[» ipf the new overlay. 

Host of the utilities update or read the fey\stem^ relations usinr^ 

me wh: 



AMI calls • MODIFY contains a sort routine 



Lch puts tuples in 



collating s'equenc^acccxrding to the concatenation of the desired 
keys (whicn need not bK^of tne samfe data type)J^ Pages are inir. 
tially loaded to approximately 80^5 of capacity. The sort ro-utine 
is a recursive N-v/ay rherge-sortN^nere N is maximum number of 
files process 4^anmave open at o^qe (currently 8). The index 
builaing occurs in an obvious way. To cobv^rt to hash st-ructures 
hODIFY must specify the number of primary pap:e|x^o be allocated. 
This parameter, is used* by the AMI in its hash ^^heniie"'T^7trir^^ 
standard modulo division~metnod ) . This is d^^e by a rule of 
thumb. 




It should De noted that a user who creates an empty hash relation 
using tne CREATE commar.d and then ^copies a large UNIX file into 
it using COPY v/ill create^ a very inefficient structure. This is 
because a relatively small default number of primary pages will 
nave been specified by 'CREATE and overflow chains will be lonrr. 
A better strategy is to COPY into an . unsorted table so that 
MODIFY can subsequently make a .good guess at tt^e number of pri- 
mary pafjes to allocate 



' 6.2 '^DEFEFRED UPDATE AND RECOVERY 



Any updates (APPEND, DELETE,^ REPLACE) are processed by , writinrr 
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the tuples to be added pnanged or mbdified into a temporary ftie.. 

\ ■ / N ■• • "• 

w'nen process 3 finishes ,\ it cal/ls process 4 to actually perform 
,the modifications requested /as a final s"^e.p in processinrr^' De- 
ferred update is done for\ fo/r reasons. 

a) Secondary index corl/iderations . Suppose the follb^lnc: QUEL 
statement is executed; ' 

RANGE OF E IS EMPLOYEE 

REPLACE E(SALARY = 1 . 1«E. SALARY) WHERE 

. E. SALARY /> 20000 " 

ouppose further tnat there is a secondary index on the salary 
donaln and tne primary relation is keyed on another domain. 

OVQP in findinr; the employees who qualify for the raise \;ill use 
tne secondary index. If one employee (say Smith qualifies and 
his tuple is noi^ified and the secondary index updated) then the 

scan of tne sec^or.dary index will find his "tuple a seconcl tine (in 

/ 

fact an arbit^r^ary number of times). Either secondary indexes 
cannot be u^ed to identify qifj^ifyinrj^ when ranrje qualifi- 

cations are present (a rat her^unnktljral restr^Kiiaon ) or secondary 
indices mUst be upd.c\ted in ^deferre^d mode. 

" ' ' / ^ " " \ 

b) Primary relation considerations. Suppose the followinn; OllEL 

state^iien^t is executed , 

. RliGE .OF E, M \S EMPLOYEE § y/- 

— REPLACE E(SALARY = . 9*E.S/flLAHY)^HERE ' 
: E. flGR = M.NAIIE ^ \ 

' AND • \ . c ^ 

K.3ALAHY > H. SALARY 



for, the following! EMPLOYEE relation 



NAME SAL MANAGER 

Smith 10k ^ Jones 
» . ■. . Jones Qkr 

\ Brown 9.5k - "^^-^mith 

Logically Smith should ^get the pay cut but Brown should not. 

However, "^if Smith 's, tuple is updated before Brown is checked for 

tHe pay cut, Brown. will qualify. This undesirable situation must 

'be avoided by deferred upcjate . .* 

c) Functionality of updartes. Suppose tne following "^UEL state- 
ment is executed. , . lit^^ 

RANGE o-f E,ti is EflPLOYEE ' 
REPLACE E(SALARY = M. SALARY) ' 

This update attempts to assif^n to each employee the salary of 
-^very other employee, *i.e.^ a sin/jle data item is to be replaced 
by multiple values. Sratecl differently, the REPLACE statement 
does not specify a function. This non-f uno tiona3 ity can only be 

r ■ ^ 

checked if deferred update is performed. 

d) Recovery is easier. The deferred update file provides a lo,^r^ 
of uffda|;es to be. made-* Recovery is provided upon system crash by 
the RESTORE command. In this case the deferred update routine is 
requested to destroy the, temporary file'^f it has onot yet started 
processing it. If it. has. begun processinrj, it reprooess^es the 
entire update file which is done in such a way that the effect is 
the same as if it were processed exactly once from. start to fin- 
isti. 

Hence the update is "backed out" if defe'rr'ed updQtin.f!; Has not yet 
begun; otherwise it is processed to conclusions The. software is 
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designed so the update file can be optionally spooledjMnto tape 
and recovered front tape* Ttiis ,?t?ldecl- feature should^soon ty 
operational. n ' 

.If a us^r from the terminal monitor (or a C-prof?ram) yisnes _to. 
stop a command ne can issue a "break" character, ' in this case 
all processes reset execept the deferred update .prop^ram which' 
recovers in the sane manner as above*.- o"*- 

All update cbnmands do deferred update; however th^' IHGPES utili- 
ties have not yet been modified to Mo likewise. When this is 
completed inCfiES will recover frani^ all crashes whib:h' leave the 
disk^ ^ intact . In the. meaintirne there can be disk-intact crashes 
Which cannot be recovered in this manner (if tiiey happen in such 
a way that the system catalogs are left inconsistent)'. 



The INGRES "sup^r-user" can checkpoint a daVa base(3) onto tape 
using the UIJIX backup scheme.* -Since INGRES lo^s alV Interac- 
tions, a consistent system can always be obtained (albeitN^lowly ) 
by restorinr: the last checkpoint and' running; the lor^ of ii^rac^ 
tions (or tne tape(s) of deferred updates if it exists). 



.7 CONCLUSION AHD FyfUR* EXTENSIONS . • ~ ; ^ , 

Ttie' sysrSm. described nerein is in use at 5 ^ installations and is 
.being brojJ^l:)t up at 8 others^ It for^ijrs the basis of an account,-, 
ing system,^. a system for managing .student ^ records, j^eo-data 
system^, ^^a system for naintainlf/c? a wirinrr diagram for a lar^e 
telelphone cqmpany and assorted other smaller applications.' 
Tne^se applications have been' runn^Ag for period^ of up to nine' 
montns^ ^ » ' . . 

7.1 PERFORMANCE . ' 

\ > 

At tnis time no detailed performance measurements have been made. 
However, on our system (an 11/4^0 mainframe witn 80k words oV 
core) about 4-.^6 Simult^dneous INGRES users can be supported v;ith 
reasonable response* time .(assuming they are^ doinf^ , interact ions 

whic^h affect a small^ number of tuples and for wn^ch a fast access 

\ 

path exists*. Of course^, a user has the ability to execute in- 
teraction,s in INGRES which require ' hours' to . process). This 
hardv>fare configuration costs about $60,000.^ Lart^er 11/70 instai^-^^ 
lations should be able to run substantially more "INGRES user^. 




the sizes of the<s^rocesses in INGRES are indicated below. Since 
the access methods arp loaded with processes 2 and 3 and with 
many of the utilities their 'contribution to the respective pro- 
cess sizes has' been notejd separately. . " ' ' " . 



access method^^pP') 11 K (bytes) 

terminal monitor^ ^ 10 r 



uc 





EQU£L , 
process' 2 
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process 3 (query processor)^ 
' util^T;ies- (y overlays)'' 



30 K + AH 

i)5 k'v'.+- AM 



•7.2 ^ USER FEEPACK 



Tn^feeabacR {from internal /and external uis'ers has been overy/ftelin- 
ingly positive. ^In this ,§ection Indicate features that have 
been sug(>este<l'* for future systems. - 



a)N hifjher performance ^. " 3^ 



Earlier versions of . INGRES were^ very slow.. The current versic^n- 
/sjiould .allevi^ate ^this problem . - \ 



b)'**' recursion ' 



QUEL ♦does not' support recursion. J;^nce, recursion must be' tedi- 
ously proRramned in *C usin^j the .f^reeompiler. This has been su.^- 



r.^^Sf^d as 'a;Cjesired e:<bensipn . 



c) otner^r^nf:uaf;e/ex tensions 



1 
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These inc^ofi^ user defined functions (especially counters), mul-« 
tipl^tarfetK^lists for^^a singl^ Qualification ^tatement and If- 
biien-else control structures in QUEL. 'These extensions 'are all 
so a user can avoid usin^: th'e preconpilef. * / 

d) . report ^enr^rator 
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PRJUT is orly a very , primitive report generator* " The reed foJj. . 
au^^nented facilities in thia area is clear .'^^ It. should be written 
^in lld^UEL* ' \, » -^^'^^t . . 

e) • bulk copy . , ' / • • • - 

The CpPY rpuDine fails to handle easily all situations that have , 



' Noted' throughout/ the p'aper. are a/'sas where systen irapro'vement is 

m p-rqrjress, plan-ned or desired by. users. Other areas of exten- 

/ - ^ . ' 

sion Include tii4 following ' ^ . 

a) A nulti-cc^mputer system vterslon of INGRES to operate on dis- 



tributed data bases 

•J 



b) Furtuyer performance enhancements * - - 

c) A niRher level us'er lanj^uar^e inclpdirVfT recursion and user 
defined functions ^ • 

d) . ^Dettei' .data defir.i!fion and "integrity features 

e) it data base administrator advisor. This pro^^ran would. run' at 
idle priority and issue queries ac^ainst a statistics relation to 
be. kept by IIJGRCS. It c6uld then of^fer advice to a DBA concern- 
inn the choice of access methods ancf the selection of indices. 
Tnls topic iA discussed further in [HELD75b]. 
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